Are Inductive Learning Algorithms Safe in a Continuous Learning Environment?

As intelligent software agents are pushed out into the real world and increasingly take over responsibility for important tasks, it becomes important that there are ways of guaranteeing that the agents are safe. But what does “safe” mean in this context. Certainly if the agents are interacting with people then to a certain extent, “safe” will mean that people can trust their model of the agent to be reliable and that the future behavior of the agent is predictable. Specifically, if the agent exhibits learning then the user needs to be able to predict how that learning affects the agent’s future behavior. Right now, the main learning entity that most people interact with are other people. At least initially, people will assume that learning agents’ behavior will have many of the same properties as shown by human learning behavior. In this paper we will look at one such property, learning’s preservation of competence. People assume that learning improves performance, rather than degrades it. In particular, people assume that if someone knows how to solve a particular problem and then learns more about that domain, that they will still be able to solve that problem. Occasionally, learning interferes with problem solving performance and this might result in the learner taking longer to solve the problem than before. However, we normally expect that the learner will still be able to solve the problem1. This places an important constraint on agent learning behavior. People will be unwilling to work with learning agents if the agent’s learning makes its behavior unpredictable. Specifically, learning should not decrease the range of problems that an agent can solve.